1. Field of the Invention
The present invention relates to an image processing apparatus and method.
2. Description of the Related Art
In recent years, a solid-state imaging device, such as a charge coupled device (CCD) sensor and a complementary metal oxide semiconductor (CMOS) sensor, has been used as an imaging device of an image inputting apparatus, such as a television camera, a video camera, and a digital camera. However, the dynamic range of the solid-state imaging device is narrower than that of a film of a film-based camera, and the image quality of the solid-state imaging device deteriorates in some imaging conditions.
Accordingly, as a method of expanding the dynamic range of the solid-state imaging device, there is a wide dynamic range image synthesizing technique using a multiple exposure system, in which a plurality of images of the same scene, which images have different exposure quantities different from one another, are photographed and the photographed plurality of images are synthesized to obtain an image having an expanded dynamic range. The flow of the image synthesizing processing will be described step by step below.
(1) In order that the image signals on the lower luminance side of a subject may take suitable values, a long second exposure image accumulated for a long exposure time is imaged. However, in the long second exposure image, the quantity of light on the higher luminance side of the subject is large, and the “saturation into white,” in which an image signal is saturated, occurs.
(2) A short second exposure image, which has been accumulated for a short time during which the image signals on the higher luminance side of the subject take suitable values, is imaged. An image at all the imaged pixels of which image signals are not saturated to cause no “saturation into white” is obtained.
(3) The pixel signals from the long second exposure pixels at which the “saturation into white” occurs are abandoned, and the pixel output values of a short second exposure image at the coordinate positions of the abandoned image signals replace the abandoned image signals.
(4) A correction gain G is determined on the basis of a ratio of each of the pixel output values of the long second exposure image and the short second exposure image, and the replaced short second exposure image is multiplied by the determined correction gain G to correct the shifts between the luminance levels of both the images.
If the ratio of exposure times is made to be the correction gain G, then offsets remain in the boundary parts between the long second exposure pixel output values and the short second exposure pixel output values, and the offsets are visually observed as a fake edge. The cause of the occurrence of the fake edge is that the correction gain quantity necessary for the correction is not only determined by the ratio of the exposure times, but also is influenced by the nonlinearity of the outputs of the sensor, the non-linear gain errors of a reproducing analog circuit system, and the like, besides. Accordingly, the correction gain G is calculated on the basis of real image information, as described with regard to the aforesaid item (4).
A method of calculating the correction gain G from the long second and short second exposure pixel output values is disclosed in the following first patent document. The correction gain determining method of Japanese Publication of Patent No. 3420303 (the first patent document) first calculates the ratio of luminance values at each of the pixels of an imaged long second exposure image and an imaged short second exposure image. The calculated ratios are expressed as luminance level ratios. The method next calculates an average of the luminance level ratios of a pixel group having the same luminance value of the long second exposure image (hereinafter referred to as long second luminance value) on the basis of the long second luminance values, and the calculated average of the luminance level ratios is set as a luminance level ratio representative value at the long second luminance value. Then, the method produces a graph having an X-axis plotted by long second luminance values and a Y-axis plotted by the luminance level ratio representative values. Because the long second exposure pixels cause the saturation into white in an area in which the long second luminance values (X-axis) are large and the short second exposure pixels cause black sinking in an area in which the long second luminance values (X-axis) are small, the luminance level ratio representative values (Y-axis) in these areas have large errors from the values to the long second luminance values (X-axis) near to the center of them, and the produced graph becomes a curve having a gentle peak at an intermediate level of the long second luminance values (X-axis). The range of use of the long second luminance values (X-axis) to be used for calculating the correction gain G is determined around the long second luminance value (X-axis) corresponding to the peak position. The luminance level ratio representative values (Y-axis) corresponding to the long second luminance values (X-axis) situated in the set range are extracted, and the average value of the extracted luminance level ratio representative values (Y-axis) is calculated to determine the correction gain G.
However, the determination method of the correction gain disclosed in the aforesaid first patent document has a point requiring examination: the method must execute enormous calculations. In order to calculate the correction gain, the following mass operations are necessary: division operations for obtaining the luminance level ratios to all of the pixels, the sorting processing and the product-sum operations for totalizing the calculated luminance level ratios for every long second luminance value to obtain the average of them, and the like. If the calculation method is realized by software or hardware, a long time is needed for the calculation up to the correction gain determination. In particular, in the case of a camera for a moving image (monitoring camera) installing a wide dynamic range image synthesizing processing function, the correction gain determining method disclosed in the first patent document is required to end the calculation of the correction gain within 1/60 seconds if the frame rate of the camera for a moving image is 60 fps. However, the hardware of the camera for a moving image has not been able to realize the completion of the calculation within 1/60 seconds. Then, the calculation of the correction gain from image signals has not been performed, but the wide dynamic range image has been synthesized by using a coefficient determined beforehand. Consequently, fake edges caused by the errors of the correction gains have occurred in final wide dynamic range images.
Moreover, because the technique disclosed in the first patent document adopts the method of determining the correction gain on the basis of the calculation results of the luminance level ratios to all of the pixels, the number of pixels to be used for the calculation of the correction gain is uniquely determined. In order to improve the calculation accuracy of the correction gain and to synthesize a wide dynamic range image having an ultra-high image quality emphasizing a fake edge reducing effect, it is advantageous to have a large number of pixels to be used for the calculation as many as possible. However, in the application sufficient for a normal image quality, it is over specification to have the ultra-high image quality caused by improving the calculation accuracy of the correction gain.
Because the technique disclosed in the first patent document adopts the method of determining the correction gain on the basis of the calculation results of the luminance level ratios to all of the pixels, the method cannot set the number of the pixels to be used for the calculation to an arbitrary number. Consequently, it is impossible to flexibly set the correction gain calculation accuracy and the calculation speed according to each application.
It is an object of the present to provide an image processing apparatus and method, both capable of determining a correction gain factor to synthesize an image at a high rate of speed.